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Executive  Summary 

The  Department  of  Defense  (DoD)  has  placed  a  growing  emphasis  on  the  pursuit  of  agile 


capabilities  via  net-centric  operations.  The  breadth  of  technological  advancements  in 
communication  and  sensing  has  generated  exciting  opportunities  for  battlefield  systems  to 
exploit  collaboration  to  multiple  effects.  In  this  setting,  systems  able  to  interoperate  along  several 
dimensions  increase  the  efficiency  of  the  overall  system-of-systems  (SoS)  manifold.  However, 
the  manner  in  which  these  SoS  are  acquired  (designed,  developed,  tested,  and  fielded)  hasn’t 
completely  kept  pace  with  the  shift  in  operational  doctrine.  In  our  current  project,  we  have 
attempted  to  unravel  the  layers  of  complexities  in  an  SoS  acquisition  program,  outline  an 
acquisition  strategy  better  suited  for  such  programs,  and  develop  an  exploratory  analysis  tool  to 
provide  insights  into  the  acquisition  process. 

The  research  efforts  during  the  one  year  study  period  have  focused  on  the  development  and 
consequent  extension  of  prior  frameworks  of  the  Computational  Exploratory  Model  (CEM)  — a 
discrete  event  simulation  model —  and  its  associated  analytical  representation,  the  Markov 
approach,  to  investigate  the  impact  of  development  dependencies  on  the  successful  acquisition  of 
an  SoS.  These  efforts  also  include  a  complementary  decision  analysis  tool  that  is  based  on 
investment  portfolio  theory.  The  conceptual  model  for  acquisition  strategy  proposed  in  our 
project  is  based  on  the  16  technical  management  and  technical  system-engineering  processes 
outlined  in  the  Defense  Acquisition  Guidebook  (DAG),  often  referred  to  as  the  5000-series 
guide.  Our  conceptual  model  for  acquisition  is  centered  on  the  revised  processes  of  the  2007 
System-of-Systems  System  Engineering  (SoS-SE)  manual.  The  Markov-based  analytical 
approach  seeks  to  augment  the  CEM  approach  by  developing  a  method  that  enables  the 
comparison  of  networks  of  systems  that  are  interconnected  and  quantifying  the  cascading  effects 
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of  development  risk.  The  goal  of  this  research  work  is  to  allow  acquisition  professionals  to 
develop  intuition  for  procuring  and  deploying  an  SoS.  The  investment  portfolio-based  approach 
pursues  this  goal  by  providing  a  means  of  objectively  choosing  baskets  of  systems  that  comprise 
an  SoS,  which  balances  performance  against  developmental  cost  and  risk  from  resulting  system 
interdependencies. 

Applications  of  the  Markov  network  analysis  method  and  investment  portfolio  approach 
enables  the  identification  of  optimal  SoS  network  topologies  and  provides  a  tool  for  acquisition 
professionals  to  identify  the  features  of  an  SoS  that  contribute  most  to  the  successful  progression 
(or  delays)  in  the  development  process. 

This  report  summarizes  the  progress  made  during  the  previous  year’s  work  that  enhanced 
the  ability  to  address  systems-specific  risk,  its  propagation  to  interdependent  systems,  and  the 
comparison  of  SoS  alternatives.  This  analysis  of  alternatives  is  made  possible  by  both  the 
Markov-based  model  as  well  as  the  investment  portfolio-based  optimization  approach  developed 
by  the  researchers  during  this  period.  Example  studies  presented  in  this  report  illustrate  both  the 
Markov  approach  and  investment  portfolio  approach  for  constructed  sample  problems.  The 
Markov  analytical  model  is  demonstrated  for  a  conceptual  coffee  maker  example  (summarized 
from  our  published  journal  article).  The  investment  portfolio-based  method  is  applied  to  an 
acquisition  scenario  for  the  Littoral  Combat  Ship  (LCS)  initiative. 

Outreach  &  Collaboration 

Our  work  during  the  project  period  has  resulted  in  one  external  journal  publication,  one 
conference  paper/presentation,  and  one  submitted  conference  paper  (for  the  2012  NPS 
Acquisition  Symposium). 
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First  and  foremost  was  a  journal  article  submission  that  was  accepted  and  published  in  the 
ASME  Journal  of  Mechanical  Design  (Mane  et  al,  2011).  The  article  included  updated  work  that 
extends  the  Markov  perspective  to  also  include  time-varying  elements  of  the  Markov  model.  The 
article  and  associated  research  received  valuable  feedback  through  the  review  process  and 
provided  additional  insight.  The  feedback  included  suggestions  for  transition  probability 
estimations  and  issues  on  scalability  of  the  method  to  larger  sized  problems.  Second,  we 
delivered  an  earlier  conference  paper  and  presentation  on  the  Markov  approach  at  the  IEEE 
System,  Man,  and  Cybernetics  (SMC)  Conference  in  Anchorage,  AK,  in  October  2011. 

Feedback  from  both  journal  article  reviewers  and  presentation  attendees  has  enhanced  the 
knowledge  base  developed  for  further  proliferation  of  the  Markov-based  model  in  this  research. 
Third,  we  submitted  the  investment  portfolio  work  for  presentation  and  proceedings  publication 
at  the  upcoming  NPS  Acquisition  Research  Symposium,  May  15-17,  2012,  in  Monterey,  CA. 

We  anticipate  that  the  contributed  work  will  warrant  further  collaboration  and  additional 
discourse,  particularly  with  members  of  the  NPS  community  who  conduct  research  in  portfolio 
methodologies  and  investment  valuation  research. 

Additionally,  we  have  identified  key  figures  both  in  the  NPS  and  PEO-LCS  communities 
who  we  feel  are  instrumental  for  collaboration  in  the  ongoing  research  work.  The  identified 
figures  include  current  lead  system  integrators  for  the  Navy’s  LCS  endeavor  and  draws  potential 
for  further  enrichment  of  the  current  portfolio  example  that  is  based  on  current  LCS  efforts.  We 
intend  to  incorporate  an  added  dimension  of  realism  to  the  currently  developed  methods  through 
collaborations  with  these  connections  and  receiving  direct  feedback  from  real-world  acquisition 
practitioners.  This  will  serve  as  a  rich  basis  for  further  ideations  and  the  development  of  the 
portfolio-based  approach  into  a  mature  decision-making  platform. 
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Introduction 


The  purpose  of  capabilities-based  acquisition,  as  described  by  Charles  and  Turner  (2004),  is 
to  acquire  a  set  of  capabilities  instead  of  acquiring  a  family  of  threat-based,  service-specific 
systems.  The  Missile  Defense  Agency  (MDA),  for  example,  uses  capability-based  acquisition  to 
evaluate  the  success  of  a  program  based  on  its  ability  to  provide  a  new  capability  for  a  given 
cost,  and  not  on  its  ability  to  meet  specific  performance  requirements  (Spacy,  2004).  The  Joint 
Mission  Capability  Package  (JMCP)  concept  is  another  example  that  aims  to  create  a  joint 
interdependency  between  systems  to  combine  capabilities  in  order  to  maximize  reinforcing 
effects  and  minimize  vulnerabilities  (Durkac,  2005).  The  goal  of  the  JMCP  effort  is  a  more 
efficient  utilization  of  both  human-  and  machine-based  assets  and,  in  turn,  improved  combat 
power. 

To  accomplish  the  desired  capability,  systems  are  increasingly  required  to  interoperate  along 
several  dimensions,  which  characterizes  them  as  SoS  (Maier,  1998).  SoS  most  often  consist  of 
multiple,  heterogeneous,  distributed  systems  that  can  (and  do)  operate  independently  but  can  also 
collaborate  in  networks  to  achieve  a  goal.  Examples  of  SoS  include  civil  air  transportation 
(DeLaurentis  et  al.,  2008),  battlefield  ISR  (Butler,  2001),  missile  defense  (GAO,  2007b),  etc. 
According  to  Maier  (1998),  the  distinctive  traits  of  operational  and  managerial  independence  are 
the  keys  to  making  the  collaboration  work.  The  network  structure  behind  the  collaboration, 
however,  can  contribute  both  negatively  and  positively  to  the  successful  achievement  of  SoS 
capabilities  and,  even  earlier,  to  the  developmental  success.  Collaboration  via  interdependence 
may  increase  capability  potentials,  but  it  also  contains  concealed  risk  in  the  development  and 
acquisition  phases.  Brown  and  Flowe  (2005),  for  instance,  have  investigated  the  implications  of 
the  development  of  SoS  to  understand  the  drivers  that  influence  cost,  schedule,  and  performance 
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of  SoS  efforts.  Results  of  their  study  indicate  that  the  major  drivers — as  indicated  by  subject- 
matter-experts — include  systems  standards  and  requirements,  funding,  knowledge,  skills  and 
ability,  system  interdependencies,  conflict  management,  information  access,  and  environmental 
demands. 

Disruptions  in  the  development  of  one  system  can  have  unforeseen  consequences  on  the 
development  of  others  if  the  network  dependencies  are  not  accounted  for.  The  goal  of  a  single 
system’s  program  manager  is  the  mitigation  of  risk  leading  to  successful  development  of  that 
specific  system.  While  program  managers  nearly  always  consider  direct  or  immediate 
consequences  of  decisions,  they  often  don’t  consider  the  cascading  second-  and  third-order 
effects  that  result  from  the  complex  interdependencies  between  constituent  systems  in  an  SoS.  It 
falls  on  acquisition  managers  and  systems  engineers  (or  SoS  engineers)  to  understand  and 
manage  the  successful  development  of  a  system,  or  family  of  systems,  to  produce  the  targeted 
capability  in  this  challenging  setting. 

Abundant  evidence  suggests  that  SoS-oriented  endeavors  have  struggled  to  succeed  amidst 
the  development  complexity.  The  Future  Combat  System  (FCS)  is  the  most  recent  example 
(Gilmore,  2006).  Civil  programs  have  not  been  spared  either,  for  example,  the  Constellation 
Program  (Committee  on  Systems  Integration  for  Project  Constellation,  2004)  and  NextGen 
(NextGen  Integration  and  Implementation  Office,  2009).  Rouse  (2007)  summarizes  the 
complexity  of  a  system  (or  model  of  a  system)  as  related  to  the  intentions  with  which  one 
addresses  the  systems,  the  characteristics  of  the  representation  that  appropriately  accounts  for  the 
system’s  boundaries,  architecture,  interconnections  and  information  flows,  and  the  multiple 
representations  of  a  system. 
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The  research  work  our  group  has  conducted  under  funding  from  the  NPS  Acquisition 
Research  Program,  including  this  present  report,  specifically  targets  complexities  stemming 
from  system  development  risk,  the  interdependencies  among  systems,  and  the  span-of-control  of 
the  systems  or  SoS  managers  and  engineers.  The  objective  of  the  research  is  to  quantify  the 
impact  of  system-specific  risk  and  system  interdependency  complexities  using  (a)  our  evolving 
computational  exploratory  modeling  approach  and  (b)  an  analytical  Markov  approach  for 
quantifying  the  same  effectsjan  investment  portfolio  framework  for  facilitating  acquisition 
decisions.  The  work  documented  in  this  report  comprises  new  improvements  to  an  analytical 
component  of  the  CEM  previously  introduced  in  prior  Acquisition  Research  Program  Symposia 
(Mane  &  DeLaurentis,  2009,  2010).  The  aim  of  the  CEM  is  to  provide  decision-makers  with 
insights  into  the  development  process  by  propagating  development  risk  in  the  SoS  network  and 
capturing  the  impact  that  system  risk,  system  interdependencies,  and  system  characteristics  have 
on  the  timely  completion  of  a  program.  We  also  introduce  extended  work  related  to  the  Markov 
approach  to  treat  the  same  complexities  via  computations  on  conditional  probabilities  that  relate 
to  the  transmission  of  risk  in  network  dependent  systems.  Furthermore,  an  investment  portfolio- 
based  approach  complements  the  Markov  approach  with  a  decision  tool  framework  as  a  means  of 
leveraging  perfonnance  against  risk  and  cost. 
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Markov  Analytical  Approach 


Additional  complexity  in  the  model,  carefully  selected,  will  likely  increase  the  efficacy  of 
the  CEM.  However,  as  a  simulation-based  approach,  it  too  has  limitations.  Therefore,  in 
conjunction  with  the  further  development  of  the  CEM,  the  researchers  are  also  developing  an 
analytical  approach  that  captures  the  characteristics  of  a  network  that  results  from  the 
developmental  interdependencies  of  systems.  This  Markov  analytic  approach  uses  a  network- 
level  metric  to  treat  the  same  complexities  via  computations  on  conditional  probabilities  that 
relate  to  the  transmission  of  risk  in  networks  of  interdependent  systems.  The  Markov  approach 
provides  means  to  compare  networks  in  their  ability  to  arrest  the  propagation  of  delays  caused  by 
random  disturbances  and  can  be  used  as  a  figure  of  merit  when  designing  SoS  architectures  that 
aim  to  achieve  some  desired  capability. 

While  typical  networks  like  the  World  Wide  Web,  social  networks,  and  communication 
networks  are  a  result  of  evolution,  the  networks  created  by  the  development  of  interdependent 
systems  can  be  designed  to  achieve  a  desired  performance.  Being  able  to  quantify  the 
performance  of  such  networks  enables  comparison  of  networks  and,  ultimately,  the  design  of 
networks  that  optimize  that  perfonnance.  During  development,  the  ability  of  a  network  of 
systems  to  propagate  or  arrest  disruptions  can  be  an  important  performance  parameter  when 
selecting  a  family  of  systems  to  provide  a  certain  level  of  capability. 

Network  analysis  tools  can  help  to  describe  the  properties  of  a  network  and  to  identify 
critical  component  systems.  The  number  of  links  and  nodes  in  a  network,  for  instance,  can 
indicate  the  complexity  of  a  network  by  measuring  the  number  of  systems  and  their  link. 
Similarly,  network  average  degree,  which  describes  the  average  number  of  links  of  each  node, 
can  indicate  the  level  of  connectivity  in  a  network  and  help  identify  critical  systems.  These 
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traditional  network  measures,  however,  are  unable  to  describe  the  performance  of  the  entire 
network  and,  consequently,  comparison  of  networks  in  their  ability  to  arrest  the  propagation  of 
disruptions  that  can  create  development  delays. 

Delay  propagation  modeling  is  common  in  the  airline  industry,  where  delays  at  one  airport 
can  easily  propagate  in  the  aviation  network  and  impact  dependent  airports.  Approaches  for 
modeling  and  estimating  these  delays,  however,  center  on  regression  analysis  (Xu  et  al.,  2005; 
AhmadBeygi  et  al.,  2008).  AhmadBeygi  et  al.  (2008),  for  instance,  investigates  the  relationship 
between  the  potential  propagation  of  flight  delays  to  subsequent  flights  and  the  utilization  levels 
of  air  service  providers.  Even  though  the  delays  can  propagate  indefinitely,  the  delay 
propagation  structure  is  acyclic.  A  delay  caused  by  mechanical  problems  to  an  aircraft  will 
always  propagate  forward.  In  the  system  development  process,  delays  can  be  cyclic,  which 
increases  the  complexity  of  the  problem  and  limits  the  ability  of  current  approaches  to  quantify 
the  total  delay. 

This  research  presents  a  network-level  metric  that  captures  the  characteristics  of  a  network 
that  results  from  the  development  interdependencies  of  systems  and  provides  means  to  compare 
networks  in  their  ability  to  arrest  the  propagation  of  delays  caused  by  random  disturbances.  As 
previously  stated,  we  present,  in  this  paper,  an  approach  that  aggregates  the  system  and  system 
interdependency  characteristics  of  a  development  network  of  systems  into  a  single  metric — 
expectation  and  standard  deviation  of  delays  or  costs — and  enables  a  meaningful  comparison  of 
alternate  networks  in  their  ability  to  arrest  the  propagation  of  developmental  delays. 

The  Markov  approach  characterizes  the  propagation  of  delays  in  a  network  of  systems  as  a 
Markov  Process.  A  Markov  chain  is  an  indexed  collection  of  random  variables  used  to  model  a 
sequence  of  dependent  events  such  that  the  probability  of  the  occurrence  of  a  given  event  is 
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solely  dependent  on  the  previous  event  (Sheskin,  201 1).  While  the  mathematics  are  well 
developed,  we  propose  the  application  of  the  classical  lost-miner  problem  (or  gambler-ruin 
problem;  Ross,  2007)  to  the  development  of  a  network-level  metric  that  enables  comparison  of 
networks.  The  metric  measures  the  ability  of  a  network  to  arrest  the  propagation  of  delays  in 
terms  of  the  expected  total  delay  and  the  standard  deviation.  The  values  associated  with  this 
metric  captures  the  cascading  effects  of  delay  transmissions  through  transition  probabilities. 

The  delay  propagation  is  modeled  as  a  Markov  chain,  where  the  states  are  defined  as  the 
constituent  systems  and  the  transition  probabilities  as  the  dependency  strengths  between  systems. 
Of  relevance  is  the  computation  of  the  probability  of  the  passage  in  n  time  steps  from  any  of  the 
states  to  an  absorbing  state — a  state  from  which  one  cannot  exit,  is  the  computation  of  this 
probability  is  analogous  to  a  disruption  due  to  delays  being  arrested  after  n  time  steps  or  the 
disruption  being  arrested  after  a  cost  of  n  dollars.  Consider  the  development  of  a  generalized 
three-system  network  of  interdependent  systems  (see  Figure  1).  Development  of  system- 1 
(denoted  by  x/)  depends  on  the  development  of  system-3  (x?),  and  vice-versa,  while  development 
of  system-2  (x?)  depends  on  the  development  of  system- 1.  The  interconnectivity  between 
systems  implies  that  information  or  design  decisions  in  one  system  impact  the  development  of  a 
dependent  system. 

t33 


Figure  1.  A  Three-System  Development  Network 
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Assume  that  the  network  is  modeled  by  an  interdependency  matrix  T.  The  entries  of  T  are 
positive  and  correspond  to  the  conditional  probability  of  a  delay  propagating  to  system  j,  given 
that  it  is  in  system  i,  T-  =  P(x/  x,  j .  Note  that  T  is  not  necessarily  symmetric.  Modeling  this 

network  as  a  Markov  chain  implies  that  the  columns  of  T  must  sum  to  a  value  less  than  or  equal 
to  one.  In  the  implementation  of  the  PageRank  '*  algorithm,  a  damping  d  with  a  value  between 
[0,1)  is  used  to  ensure  that  (I-r/T)  is  invertible.  Motivated  by  PageRank®,  let  state  xp  represent  an 
absorbing  state  and  c  =  rFi  =  p(xF  \  x, )  represent  the  probability  of  a  delay  transitioning  from  a  state 
Xj  to  state  A'/  and  arresting  the  propagation  of  a  delay.  Note  that  c  is  defined  as  a  row  vector.  This 
is  essentially  the  damping  term  d  in  the  PageRank  8  algorithm,  and  the  transition  probability 
matrix  T  can  thus  be  defined  as 


where  the  matrix  A,  defined  as 


T  = 


A  0 
c  I 


(1) 


Ak,=p(xk(n+x)\xM) 

(2) 

represents  the  transition  probabilities  of  a  delay  being  in  state  xa  at  time  n+1,  given  that 
there  was  a  delay  inx,  at  time  n.  The  entries  of  A  contain  all  the  transition  probabilities  of  T 
except  the  transitions  to  the  absorbing  state  xp.  In  particular,  (I- A)  is  invertible  and  the 
eigenvalues  of  A  are  contained  in  the  unit  disc. 

To  compute  the  location  of  the  delays  at  any  given  time  n,  let 


%{n)=P(Xj{n)) 

be  the  probability  of  delay  at  location  Xj  at  time  n  and  let 


(3) 
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bj  =^M°)) 

(4) 

be  the  vector  indicating  the  probability  that  the  initial  delay  occurs  in  system  xj  at  time  n= 0. 
Note  that  a  delay  can  occur  only  in  one  system  at  any  given  time;  therefore,  V  bp- 1 .  This 

*—‘i  J 

quantity  can  be  described  as  the  likelihood  of  a  system  to  experience  a  delay.  Additionally,  let 

ct  =P(xF(«  +  l)|x;W) 

(5) 

be  the  row  vector  indicating  the  probability  of  a  delay  being  arrested  at  time  n+ 1 ,  given  that 
it  is  in  state  x,  at  time  n.  So  the  state  space  form  of  the  probability  of  the  location  of  delays  at  any 
given  time  is 

%{n  +  l)  =  A  c(n)  subject  to  c(0)  =  b 
f{h  |  Xj  (0))  =  c^(n)  (6) 

where  F  is  the  probability  of  a  delay  transitioning  to  the  absorbing  state  (e.g.,  the  probability 
of  a  delay  being  arrested),  given  that  it  starts  at  any  location  Xj  (see  Equation  7). 


F(/z|xy.(°))=P(xF(n)|x;(0)) 

The  probability  of  delays  being  arrested  at  time  n  can  thus  be  expressed  as 


I 


(V) 


F(n  |  Xj  (0))  =  p(xf  (n)  |  Xj  (0))  =  cA"b 


I 


(8) 


The  developed  set  of  Equations  1-8  is  the  implementation  of  a  modified  PageRank ' 
algorithm  that  can  be  employed  to  rank  the  vulnerability  or  criticality  of  the  constituent  systems 
in  a  network  of  systems.  As  previously  mentioned,  system  or  product  development  can  be 
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disrupted  by  endogenous  and/or  exogenous  factors  that  can  result  in  development  delays.  Due  to 
the  network  structure  of  the  development  process  and  system-specific  characteristics,  delays  in 
one  system  can  result  in  disruptions  to  dependent  systems,  which  can  result  in  further  delays  and 
the  propagation  of  these  to  adjacent  systems.  The  parameters  described  here  provide  a 
quantitative  representation  of  these  characteristics.  The  probability  of  the  initial  occurrence  of  a 
delay,  bj  =  p[xj  (o)j ,  can  be  seen  as  a  descriptor  of  the  risk  associated  with  the  development  of  a 

system  j.  For  instance,  if  the  design  of  system  j  involves  the  development  of  new  technologies 
with  a  low  Technology  Readiness  Level  (TRL),  the  probability  of  disruptions  occurring  during 
its  development  (i.e.,  technology  setbacks,  failure  to  meet  milestones  or  gate  reviews)  is 
expected  to  be  high.  Conversely,  if  the  design  of  system  j  is  based  on  mature  technology  or  off- 
the-shelf-parts,  the  likelihood  of  disruptions  during  its  development  process  can  be  expected  to 
be  low.  In  the  method  presented  for  the  Markov  analytical  approach,  these  probabilities  are 
nonnalized  so  that^  bt  =  1 .  The  conditional  probability  of  the  arrest  of  a  delay, 

p(xf  (n  + 1)|  Xfin)) ,  can  be  interpreted  as  the  ability  of  a  system  to  absorb  delays  (e.g.,  a  delay  in 
upstream  systems  does  not  disrupt  the  development  process  of  the  system  of  interest)  and  not 
propagate  them  to  dependent  systems;  the  disruption  can  be  a  function  of  the  size  of  the 
development  program,  its  funding,  its  schedule  slack,  or  other  technological  or  political 
attributes.  For  instance,  an  underfunded,  highly  visible,  and  controversial  program  may  be  less 
likely  to  absorb  delays  and  more  likely  to  pass  those  to  dependent  systems,  disrupting  their 
development  and  causing  additional  delays.  Alternatively,  a  well-funded  program  may  be  more 
likely  to  absorb  delays  and  not  propagate  them  to  dependent  systems  by  hiring  additional 
personnel,  acquiring  more  resources,  or  any  other  response  that  could  be  achieved  by  increasing 
expenditures.  The  transition  probabilities  in  matrix  A  can  be  seen  as  descriptors  of  the 
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dependency  strength  between  systems.  We  assume  that  methods  similar  to  the  ones  proposed  by 
Sosa  et  al.(2003)  and  Sharman  and  Yassine  (2004)  can  be  used  to  quantify  dependency  strengths 
between  systems  based  on  spatial,  structural,  energy,  material,  and  infonnation  interactions.  The 
goal  of  this  work  is  not  to  quantify  these  factors  but  to  provide  a  means  to  aggregate  these 
descriptors  to  compute  a  network-level  metric  that  facilitates  comparison  of  design  alternatives 
while  considering  both  the  direct  and  indirect  impacts  of  interactions  between  systems. 

Impact  of  Disruptions 

While  the  likelihood  of  delay  propagation  can  help  to  describe  the  performance  of  a 
network,  not  all  disruptions  are  equally  important.  The  development  risk  of  a  network  of 
systems  can  be  high  because  of  the  large  likelihood  of  disruptions  (due  to  development  delays) 
but  also  because  of  the  impact  of  those  disruptions.  In  this  section,  we  present  an  extension  to  the 
proposed  method  that  enables  the  quantification  of  risk  in  terms  of  development  delay,  which  is 
well  correlated  with  cost.  Consider  the  same  three-system  network  in  Figure  2,  but  now  assume 
that  we  can  quantify  the  impact  of  disruptions  between  any  two  systems. 


Figure  2.  Three-System  Development  Network  with  Disruption  Impact 

The  matrix  Dp  is  defined  as  the  time  delay  caused  by  a  disruption  in  system  i,  which  causes 
a  delay  and  a  resulting  disruption  to  system  j.  For  example,  system- 1  may  experience  a  budget 
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cut,  with  some  probability  pfifio)),  that  disrupts  its  development  schedule.  Four  possibilities  of 
disruption  outcomes  emerge:  (1)  the  disruption  can  have  no  impact  in  the  development  of 
system- 1  with  probability  tfi  =  p{xf  x,  ) ,  and  nothing  is  affected;  (2)  the  disruption  can  cause  a 
development  delay  of  Dn=  2  time  units  with  probability  rn  =  r(xl  \  x, )  in  the  development  of 
system- 1  that  is  not  sufficiently  large  enough  to  impact  dependent  systems;  (3)  the  disruption  can 
result  in  a  delay  of  D3i  =1  time  unit  with  probability  r31  =  p(x3 1  x, )  that  impacts  the  development  of 
system-3;  or  (4)  the  disruption  can  result  in  a  delay  of  D21  =  2  time  units  with  probability 
r2l  =  p(x2 1  x, )  that  impacts  the  development  of  system-2.  Depending  on  the  affected  system  and 
the  likelihood  of  the  delay  propagating  to  additional  systems,  this  process  will  continue  until 
state  x/.-is  reached  and  the  propagation  of  the  delay  is  arrested. 

To  detennine  the  probability  of  a  delay  being  arrested  at  time  n,  the  definition  of  the  states 
in  the  Markov  approach  is  modified  such  that  each  transition  corresponds  to  the  accumulation  of 
one  time  unit  of  delay,  while  ensuring  that  the  transition  probability  matrix  A  retains  its  Markov 
properties  and  (I- A)  is  invertible.  When  the  duration  of  the  delay  caused  by  a  disruption  is  larger 
than  one  time  unit,  additional  states  are  created  to  represent  the  one-step  transition  from  one  state 
to  the  next.  Figure  3  presents  a  numerical  example  of  this  transformation  for  the  three-system 
network. 


22 


Final  Report:  NPS  award  N00244-1 1-1-0003 
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Figure  3.  Re-Definition  of  States  for  Three-System  Development  Network 


For  instance,  because  D2i  is  of  two  time  units,  state  xs  is  created  so  that  we  now  have  D’5i 
and  D’25,  each  of  duration  one  time  unit.  Additionally,  the  transition  probabilities  are  modified 
to  accommodate  this  change  in  syntax.  For  instance,  the  probability  of  transitioning  from  state  x; 
to  state  X2  ( T21 )  is  now  represented  by  the  probability  of  transitioning  from  state  X/  to  state  X5 
( r5'j  =  p(x5  x,)=7’21 )  and  then  from  state  X5  to  state  x2  (  t25  =  p(x2  |  x5 )  =  1 ).  The  number  of  states  in 
the  modified  problem  can  be  computed  as 


+  J 


(9) 


where  dy  is  defined  as 


(10) 


and  J  is  the  number  of  the  states  (or  systems)  in  the  original  problem.  Now  the  definition  of 
the  probability  of  the  delay  being  arrested  at  time  n  can  be  computed  using  the  definition  in 
Equation  8,  where  j  e  S .  Note  that  this  approach  requires  that  the  impact  of  disruptions  be 
positive  and  represented  as  integer  values. 
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Expected  Delay  and  Standard  Deviation 

Computation  of  the  conditional  probability  of  delay  arrest  at  time  n,  given  that  a  disruption 
(due  to  a  delay)  starts  in  a  given  system  j,  provides  the  basis  for  ranking  systems  in  a  network 
based  on  their  vulnerability  and  critically  to  the  network  by  considering  the  direct  and  indirect 
impacts  of  disruptions.  The  posed  questions  are  as  follows:  What  is  the  expected  time  until  a 
delay  is  arrested?  What  is  the  standard  deviation?  Answers  to  these  questions  provide  a  means  to 
compare  the  criticality  or  perfonnance  of  individual  systems  in  a  network  as  well  as  a  network- 
level  metric  that  describes  the  entire  network  in  its  ability  to  arrest  the  propagation  of  delays. 

The  expectation  of  delay  time  provides  a  single-value  metric  while  the  standard  deviation 
provides  a  means  to  quantify  the  risk  associated  with  a  given  expectation. 

The  expected  total  delay,  given  that  a  disruption  starts  in  system  j,  is  computed  as 

E(F  |  Xj (0))  =  J  nP(xF (n)  \  Xj (o)) 

n= 0 

00 

=  YjncAnb  =  cA(l-A)~2  (11) 

n= 1 

while  the  variance  is  defined  as 

F  =  £k2|x/0))-(£k|x;(0f 

(12) 

where 

e{f2  |  Xj (O))  =  J  n2p(xF {n)  \  Xj (o)) 

n= 0 

00 

= = cA(J + AiJ  -  AVb  (13) 

n= 1 

Thus,  the  standard  deviation  is  computed  as 
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o  =  jv  =  -JcA(l  +  A){l  -  A)~3b-  [cA(l  -  A)  2bf 


(14) 


The  mean  and  standard  deviation  computation  yields  values  for  each  system  in  the  network. 
Comparison  of  these  values  with  each  other  provides  a  means  to  compare  the  perfonnance  of 
each  system  in  the  network.  Therefore,  one  can  identify  the  most  critical  or  vulnerable  system  in 
the  network  by  ranking  the  mean  and/or  standard  deviation.  This  is  essentially  the  same  approach 
used  by  the  PageRank  '*  algorithm  to  rank  the  importance,  or  relevance,  of  web  pages.  While  the 
PageRank  8  algorithm  uses  the  transition  probabilities  to  weigh  the  importance  of  links,  the 
approach  presented  here  uses  both  the  transition  probabilities  as  well  as  the  impact  of  a 
transition.  Thus,  the  criticality  of  a  system  is  measured  by  both  the  likelihood  of  propagating  a 
delay  resulting  from  a  disruption  event  as  well  as  the  quantified  impact  of  the  delay  when  the 
indirect,  propagated  consequences  are  captured.  This  provides  a  richer  understanding  of 
interdependent  systems  that  can  aid  system  engineers  and  designers  of  complex  systems  in  the 
tasks  of  identifying  critical  component  systems  and  estimating  the  potential  schedule  and  cost 
overruns  of  a  given  system  development  program 

The  ultimate  goal  of  the  research,  however,  is  to  be  able  to  compare  the  perfonnance  of 
entire  networks.  While  in  networks  like  the  World  Wide  Web,  this  question  may  not  be  relevant, 
in  the  development  of  networks  of  systems,  the  designer  has  the  option  to  design  the  network. 
Hence,  a  metric  that  enables  comparison  of  alternatives  can  be  quite  powerful.  The  metric  we 
present  in  this  section  is  the  total  expected  time  and  standard  deviation  of  arresting  a  delay.  This 
metric  is  the  weighted  sum  of  the  conditional  expectation  of  the  delay  time,  given  that  a 
disruption  starts  in  system  j: 


25 


Final  Report:  NPS  award  N00244-1 1-1-0003 


e(f)=-£e(f\Xj( o)M*,(o)) 

M 


=  (15) 


Utilization  of  this  method  to  analyze  large  problems  (e.g.,  an  increased  number  of  systems 
and  interdependencies)  hinges  on  the  ability  to  quantify  the  transition  probabilities.  As  the 
system  to  be  analyzed  grows — with  more  subsystems  and  more  dependencies — the  amount  of 
information  required  to  describe  the  transition  probability  matrix  can  be  as  low  as  the  number  of 
subsystems  (e.g.,  no  interdependencies)  or  on  the  order  of  the  number  of  subsystems  squared 
(e.g.,  all  subsystems  depend  on  all  subsystems).  But  once  the  probabilities  are  defined,  a  solution 
to  the  problem  becomes  a  matter  of  inverting  a  matrix,  whose  computational  complexity  is  O (n  ) 
for  a  fully  populated  matrix,  using  Gauss-Jordan  elimination  (Dekker  &  Hoffmann,  1989); 
therefore,  computationally,  solving  the  matrix  inverse  problem  is  the  most  “expensive”  aspect  of 
the  proposed  method.  However,  if  the  transition  probability  matrix  is  sparse  and  taking 
advantage  of  the  fact  that  its  eigenvalues  are  smaller  than  one,  a  person  could  solve  the  matrix 

inversion  problem  via  expansion  by  (/  -  a)~ 1  =  ^  Ak  . 
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Time-Dependent  Delay  Propagation 

The  approach  presented  so  far  assumes  constant  transition  probabilities.  The  probabilities  of 
delay  propagation,  however,  may  vary  with  time.  As  the  design  maturity  of  systems  increases, 
development  risk  may  decrease  or  increase  along  with  changes  in  the  interdependency  strength 
between  systems.  Similarly,  budget  and  schedule  perturbations  may  cause  changes  in  the 
probability  of  delay  propagation  in  the  later  stages  of  a  program  that  are  quite  different  from  the 
ones  at  the  beginning  of  a  program.  Therefore,  it  would  be  useful  to  be  able  to  describe  and 
compare  networks  as  interdependency  characteristics  change. 

The  proposed  Markov  approach  can  be  easily  extended  to  compute  both  the  probability  of 
delay  arrest  and  the  expected  total  delay  and  standard  deviation  as  a  function  of  changing 
interdependency  characteristics.  The  state-space  definition  now  becomes 

<^(n  + 1)  =  A(n)c(n)  subject  to  c(0)  =  b 

F(n\  Xj(0))=c(n)^(n)  (ll) 

The  transition  probability  matrix  A  and  probabilities  of  delay  arrest  for  each  system,  c,  are 
now  a  function  of  time  n  and  the  probability  of  the  delay  propagation  being  arrested  after  a  total 
delay  of  n  time  units  is 

f(h  |  xy(0))=  p(xf(/7)|  x,(0))=  c(n)f\  (18) 

t= 1 

From  this  formulation,  the  expected  delay,  total  delay  time  and  standard  deviation  caused  by 
a  disruption  due  to  a  delay  in  an  upstream  system  given  that  it  starts  in  system  Xj„  can  be 
computed.  The  difficulty  in  implementing  this  time-dependent  evaluation  of  the  perfonnance  of 
a  network  lies  in  quantifying  the  various  parameters — transition  probabilities,  probabilities  of 
disruption  occurrence  due  to  delays,  and  the  conditional  probabilities  of  delay  propagation 
arrest — as  a  function  of  time.  One  possible  approach  would  be  to  use  gate-reviews  as  discrete 
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events  that  mark  the  changes  in  risk  level  of  a  given  program.  Additionally,  one  might  estimate 
probability  of  initial  development  delay  and  disruption  occurrence  as  a  function  of  TRLs, 
perhaps  as  a  function  of  time. 

Comparison  of  Alternatives 

There  may  be  many  different  system  design  configurations  that  achieve  the  same  capability- 
level.  However,  each  configuration  represents  a  different  network  comprised  of  different 
subsystems,  different  interdependencies,  and  different  interdependency  strengths.  Among  all 
these  alternatives,  which  option  minimizes  the  risk  from  disruptions  that  can  result  in  delays?  For 
illustration  purposes,  we  present  an  example  application  of  the  proposed  Markov  approach  to 
compare  networks  based  on  a  preliminary  case  study  by  Asikoglu  and  Simpson  (2010). 

Asikoglu  and  Simpson  (2010)  consider  a  number  of  electro-mechanical  household 
appliances  to  present  their  electric  circuit  analogy  method,  by  which  they  are  able  to  rank  the 
components  of  a  product  based  on  their  vulnerability  to  design  changes.  We  use  their  example 
since  it  provides  a  basis  for  comparison  of  the  proposed  method.  In  their  coffee  maker 
application,  they  identify  its  16  components,  develop  a  Design  Structure  Matrix  (DSM),  and 
quantify  the  interdependency  strengths  between  components  by  using  the  Module  Complexity 
Score  (MCS)  of  component  interfaces  to  develop  a  weighted-MCS  that  they  employ  in  the 
electric  circuit  analysis. 

We  consider  the  same  problem  and  pose  it  in  the  context  of  development  time.  We  assume 
that  the  development  of  the  coffee  maker  is  divided  between  different  design  teams  that  are 
dependent  on  each  other  for  the  development  of  the  coffee  maker.  We  interpret  the 
interdependency  weights  between  the  coffee  maker  components  defined  by  Asikoglu  and 
Simpson  (2010)  in  the  weighted-MCS  as  the  interdependency  strengths  that  indicate  the  relative 
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difference  in  the  likelihood  of  each  component  propagating  its  delays  to  dependent  components. 
We  use  the  weighted-MCS  matrix  to  compute  the  transition  probability  matrix  Ty  by  normalizing 
the  entire  matrix  by  the  sum  of  its  entries.  By  this  process,  we  are  using  the  data  gathered  and 
analyzed  by  Asikoglu  and  Simpson  as  an  indicator  of  the  relative  likelihood  of  delay 
transmission  between  systems.  Components  that  are  connected  by  multiple  types  of  interfaces 
and  have  more  interfaces  will  have  a  higher  probability  of  transmitting  delays  to  dependent 
systems.  This  is  a  reasonable  assumption  because  tighter  coupling  between  components  increases 
the  likelihood  of  disruptions  in  one  component  having  an  impact  on  dependent  components. 

Furthermore,  we  assume  that  the  disruption  impact  is  of  one  time  unit  for  all 
interdependencies  (e.g.,  Dp  has  entries  of  one),  that  all  components  are  equally  likely  to  arrest  the 
propagation  of  a  delay  with  probability  0. 1  (e.g.,  TFi  =  0. 1),  and  that  all  components  are  equally 
likely  to  experience  an  initial  disruption  that  results  in  a  delay  (e.g.,  p(x/(o))=  j/^  =  j^6).  Note  that 

because  the  electric  circuit  analogy  does  not  include  all  these  attributes,  a  comparison  of  the 
performance  of  the  two  methods  (electric  circuit  analogy  versus  the  approach  proposed  in  this 
section)  is  not  expected  to  be  identical.  Having  generated  T,j  and  D,,,  we  can  easily  compute  the 
expected  delay  and  standard  deviation  for  this  coffee  maker  network  of  components  and  identify 
the  (direct  and  indirect)  vulnerability  to  developmental  disruptions  via  network-level  metric. 
Table  1  presents  these  results  along  with  a  comparison  against  the  change-resistance  (CR)  and 
the  weighted-MCS  results  presented  by  Asikoglu  and  Simpson  (2010). 
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Table  1.  Component  Criticality  Ranking  and  Comparison 


Change 

Resistance 

Component  Ranking[Error! 

Bookmark  not 
defined.] 

Weighted- 

MCS 

Ranking 

[Error! 
Bookmark 
not  defined.] 

Our 

Approach 

Ranking 

Our  Approach 
[Exp,  Std  Dev] 

Component- 1 

9 

8 

8 

[0.0030,  0.0597] 

Component-2 

6 

5 

5 

[0.0044,  0.0716] 

Component-3 

12 

11 

11 

[0.0010,  0.0324] 

Component-4 

16 

15 

15 

[0.0002,  0.0131] 

Component-5 

13 

12 

12 

[0.0010,  0.0321] 

Component-6 

11 

10 

10 

[0.0013,  0.0387] 

Component-7 

7 

14 

14 

[0.0003,  0.0190] 

Component-8 

1 

1 

2 

[0.0139,  0.1326] 

Component-9 

3 

3 

3 

[0.0085,  0.1079] 

Component- 10 

4 

4 

4 

[0.0073,  0.0992] 

Component- 1 1 

2 

2 

1 

[0.0141,  0.1367] 

Component- 12 

8 

6 

6 

[0.0036,  0.0676] 

Component- 13 

10 

9 

9 

[0.0016,  0.0416] 

Component- 14 

14 

13 

13 

[0.0005,  0.0221] 

Component- 15 

15 

16 

16 

[0.0002,  0.0127] 

Component- 16 

5 

7 

7 

[0.0035,  0.0681] 

Total 

[0.0040,  0.2806] 

Concerning  which  components  are  most  vulnerable,  our  method  reaches  similar  conclusions 
to  the  two  approaches  presented  by  Asikoglu  and  Simpson  (2010).  The  ranking  values  presented 
in  Table  1  indicate  which  components  contribute  most  to  the  expected  total  delay  and,  hence,  are 
the  most  critical  components  in  the  coffee  maker.  The  coffee  maker  example  assumes  that  all 
interdependencies  are  undirected  (e.g.,  the  weighted-MCS  matrix  is  symmetric).  While  this  may 
be  true  for  the  coffee  maker  example,  other  systems  may  have  directed  dependencies  or 
dependencies  that  have  asymmetrical  magnitudes.  For  instance,  the  design  of  the  housing  in  a 
coffee  maker  may  impact  the  design  of  the  carafe  more  than  the  design  of  the  carafe  may  impact 
the  design  of  the  housing.  Our  approach  can  model  this  type  of  interdependency  as  well  as 
provide  a  network-level  metric — in  terms  of  expectation  and  standard  deviation — that  evaluates 
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the  entire  network  in  its  resilience  to  delay  propagation.  This  modeling  of  network 
interdependencies  can  be  useful  if  one  is  interested  in  the  direct  comparison  of  two  or  more 
designs. 

To  demonstrate  this  network-comparison  ability,  we  consider  an  alternate  coffee  maker  that 
has  only  eight  components  (e.g.,  J  =  8)  and  which  has  a  combination  of  directed  and  undirected 
dependencies,  and  wish  to  compare  the  perfonnance  of  the  two  designs.  The  components  of  this 
design  are  presented  in  this  section  as  groupings  of  the  components  in  the  original  coffee  maker. 
We  construct  the  weighted-MCS  matrix  by  combining  the  values  of  the  original  16-component 
weighted-MCS  matrix  and  use  this  asymmetric  network  to  generate  the  transition  probability 
matrix  by  normalizing  the  entries  of  the  modified  weighted-MCS  matrix  by  the  sum  of  it  entries. 
The  nonnalization  of  entries  results  in  the  probability  matrix  presented  in  Table  2.  For  instance, 
the  probability  of  a  delay  in  the  development  of  Component-6,8,12  transitioning  (propagating)  to 
Component- 1 1  and  causing  a  disruption  is  0.2235,  while  the  probability  of  a  delay  in  the 
development  of  Component- 1 1  propagating  to  Component-6,8,12  is  0.0930. 

Table  2.  Transition  Probability  Matrix  for  Alternate  Coffee  Maker  Design 


P(Tji) 

Component- 1 

Component-3,4 

Component- 

6,8,12 

Component- 13 

Component- 1 1 

Component- 1 5 

Component- 10 

Component-9 

Component- 1 

- 

0.131 

0.0902 

0.0074 

0.0301 

0.0065 

Component-3,4 

0.0131 

- 

Component- 

6,8,12 

0.0902 

0.0930 

0.0372 

Component- 13 

0.0065 

- 

0.0065 

Component- 1 1 

0.2235 

- 

Component- 15 

0.0065 

- 

Component- 10 

0.0301 

0.0800 

- 

0.0800 

Component-9 

0.0065 

0.0080 

- 
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Furthermore,  we  assume  that  all  eight  components  are  equally  likely  to  experience  a 
disruption  (e.g.,  p(x,-(o))  =  =  %  )•  The  question  we  pose  is,  which  architecture  is  best  in  the 

context  of  development  time  if  the  alternate  coffee  maker  design  has  the  dependency 
characteristics  of  Table  2  (e.g.,  expected  delays  and  standard  deviation  due  to  development 
disruptions  caused  by  the  occurrence  and  propagation  of  delays)?  Table  3  presents  the 
expectation  and  standard  deviation  of  the  total  delay  time  for  the  alternate  coffee  maker  along 
with  a  ranking  of  the  importance  of  each  component. 


Table  3.  Performance  of  Alternate  Coffee  Maker  Design 


Component 

Alternate  Coffee  Maker 
[Exp,  Std  Dev] 

Ranking 

Component- 1 

[0.0224,  0.1783] 

4 

Component-3,4 

[0.0019,  0.0513] 

6 

Component- 

6,8,12 

[0.0339,  0.2203] 

2 

Component- 13 

[0.0018,  0.0463] 

7 

Component- 1 1 

[0.0355,  0.2301] 

1 

Component- 15 

[0.0008,  0.0292] 

8 

Component- 10 

[0.0282,  0.1957] 

3 

Component-9 

[0.0132,  0.1370] 

5 

Total 

[0.0172,0.4427] 

The  total  expected  delay  of  this  alternate  design  is  0.0172,  higher  than  the  total  expected 
delay  of  the  original  coffee  maker.  This  value  of  expected  delay  indicates  that,  for  the  assumed 
transition  probabilities,  the  original  coffee  maker  design  outperforms  the  alternate  design,  even 
though  the  latter  contains  half  the  number  of  components.  Therefore,  the  interdependency 
characteristics  of  this  seemingly  less  complex  product  make  it  a  less  desirable  design  from  a 
development  perspective.  Furthermore,  the  standard  deviation  adds  to  this  poor  perfonnance, 
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indicating  a  higher  risk  and  less  certainty  in  the  expected  performance  when  faced  with 
developmental  disruptions  due  to  delays. 

Although  a  simple  example,  this  network  comparison  brings  forth  the  difficulties  and  lack  of 
intuition  in  quantifying  complexities  of  interdependent  systems.  While  the  method  relies  on  the 
estimation  of  several  parameters,  it  does  provide  a  means  to  quantitatively  compare  networks  of 
interdependent  systems. 
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Investment  Portfolio  Approach 

A  key  component  of  our  proposed  research  under  this  project  was  the  development  of  an 


ability  to  balance  capability  and  risk  in  acquiring  systems  in  an  SoS  context.  Research  work 
accomplished  in  this  phase  and  described  in  this  report  section  centers  on  an  investment 
decision-making  tool  that  provides  a  means  of  balancing  capability  development  against  cost  and 
interdependent  risks  through  the  use  of  Modem  Portfolio  Theory  (MPT).  Prior  works  from 
surveyed  literature  have  seen  the  successful  use  of  portfolio  management  techniques  to  address 
strategic  level  asset  acquisition  and  are  extendable  to  include  multi-period  considerations.  Real 
options  analysis,  for  example,  has  been  shown  to  be  effective  and  is  widely  used  across  various 
industries  to  evaluate  discrete,  long-term  investment  strategies.  Komoroski  (2006),  for  example, 
has  developed  a  methodology  that  addressees  strategic  financial  decisions  through  an  eight-phase 
process  using  a  toolbox  of  financial  techniques — including  portfolio  optimization  techniques. 
These  frameworks  are  geared  toward  financial  uncertainty  considerations  of  strategic  projects 
and  do  not  explicitly  address  technical  architecture  and/or  evolving  SoS-wide  capabilities. 

The  investment  portfolio  approach  this  section  does  not  attempt  to  replace  but  rather 
complements  existing  methodologies  by  more  directly  addressing  issues  of  integration  and 
acquisition  from  a  robust  portfolio  theory  standpoint.  Robust  methodologies  have  been  widely 
used  by  financial  engineering  practitioners  to  manage  portfolios  in  the  face  of  market  volatility 
and  uncertainties.  The  developed  approach  in  this  section  is  also  aimed  at  improved  means  of 
performing  acquisition,  integration,  and  development  decisions  while  maintaining  advantages  in 
balancing  systems  acquisition  against  evolving  capability  requirements.  The  research  work  in 
this  section  also  addresses  more  recent  efforts  in  acquisition  that  have  emphasized  the 
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implementation  of  open  architectures  and  modularity  to  facilitate  competition  (to  lower  cost)  and 
innovation. 

Open  architecture  (OA)  involves  the  design  and  implementation  of  systems  that  conform  to 
a  common  and  unified  set  of  technical  interfaces  and  business  standards.  This  form  of 
architecture  results  in  the  development  of  modular  systems  and  increases  opportunities  for 
innovation  and  rapid  development  of  new  technologies  that  can  be  readily  integrated/swapped 
into  current  architectures.  The  Littoral  Combat  Ship  (LCS)  platfonn,  for  example,  has 
recognized  the  need  for  multivendor  acquisitions  and  OA  implementations  to  ensure  greater 
technological  adaptability.  The  LCS  program  exploits  the  benefits  of  dual  award  contracting 
under  fixed  price  initiatives  (FPI),  along  with  rapid  technology  insertion  processes  and  open 
architectures,  to  fulfill  evolving  technological  and  mission  requirements  of  littoral  warfare.  The 
combination  of  dual  contracting  and  system  modularity  helps  achieve  the  necessary  cost 
reductions  while  maintaining  a  greater  degree  of  adaptability  towards  changing  mission 
requirements.  Although  the  platform  is  not,  strictly  speaking,  an  SoS,  it  nevertheless  is  an 
excellent  representative  microcosm  of  what  constitutes  an  SoS  and  carries  many  comparable 
salient  features. 
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Figure  4.  Littoral  Combat  Ship  Layout 

(“LCS:  The  USA’s  New  Littoral  Combat  Ships,”  201 1) 

The  benefits  of  open  architectures  and  competitive  contracting  are  intuitively  clear  and  have 
been  shown  to  generate  notable  cost  savings  as  exhibited  in  previous  development  projects  such 
as  joint  direct  attack  munitions  (JDAM;  GAO  2007b)  .  However,  system  integrators  and  program 
managers  are  often  faced  with  the  challenge  of  leveraging  the  potential  benefits  of  introducing 
new  and  improved  systems  against  potential  developmental  disruptions  and  cost  considerations. 
Although  the  LCS  program  had  significant  success  through  the  dual  contracting  scheme,  it  still 
experienced  cost  overruns  due  to  a  variety  of  problems.  The  problems  included  risks  from  a 
simultaneous  design  and  build  strategy  due  to  schedule  constraints,  unrealistic  budget 
expectations,  and  market  risk  from  the  greatly  increased  price  of  steel  during  the  development 
period  (O’Rourke,  2011).  There  have  also  been  revisions  in  the  requirements  of  fleet  capabilities 
and  refocusing  of  intended  capabilities  (O’Rourke,  2011). 

Concept  Acquisition  Portfolio:  Littoral  Combat  Ship  Example 

The  littoral  combat  ships  are  designed  and  developed  by  two  primary  contractors — General 
Dynamics  and  Lockheed  Martin — as  a  result  of  the  Navy’s  dual  contract  award  strategy  that 
seeks  to  minimize  costs  through  competitive  contracting.  The  ships  are  designed  to  serve  as 
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primary  units  in  close  coastal  littoral  warfare  and  take  advantage  of  modularized  onboard 
packages  (systems)  that  are  interchangeable  for  different  operational  requirements.  These 
packages  include  the  Anti-Submarine  Warfare  (ASW),  Mine  Counter  Measure  (MCM)  and 
Surface  warfare  (SUW)  packages.  More  recent  developments  have  seen  the  introduction  of  an 
irregular  warfare  package  for  assistance  and  general  support  missions.  Although  the  LCS  is  not, 
strictly  speaking,  an  SoS,  it  nevertheless  exhibits  striking  resemblance  to  one  where  the 
conglomeration  of  systems  provide  the  intended  overarching  capabilities.  The  ongoing  work  in 
this  demonstration  assumes  a  representative  acquisition  problem  using  the  LCS  acquisition  case 
where  the  objective  is  to  achieve  desired  combat  effectiveness  and  operational  capabilities  while 
minimizing  cost  and  development  risk.  The  simple  model  inputs  and  characteristics  are  described 
in  Table  4. 


Table  4.  Individual  System  Information 


System  Capabilities  System  Develop.  Acq. 


Req. 

Time 

Cost 

Weapon 

Threat 

Anti  Mine 

Comm. 

Air/Sea  State 

Air/Sea 

Comm. 

(Years) 

($) 

Strike 

Detection 

Detection 

Capacity 

Capacity 

State 

Range 

Range 

Speed 

Package 

ASW  Variable  Depth 

0 

50 

999 

0 

0 

0 

300 

3 

3000000 

Multi  Fen  Tow 

0 

40 

999 

0 

0 

0 

200 

2 

2000000 

Lightweight  tow 

0 

30 

999 

0 

0 

0 

100 

4 

4000000 

MCN  RAMCS II 

0 

0 

40 

0 

0 

3 

100 

1 

1000000 

ALMDS  (MH-60) 

0 

0 

10 

0 

0 

4 

200 

2 

2000000 

SUW  N-LOS  Missiles 

25 

0 

999 

0 

0 

0 

300 

3 

3000000 

Griffin  Missiles 

3 

0 

999 

0 

0 

0 

200 

4 

4000000 

Seaframe  Package  System  1 

0 

0 

999 

100 

0 

0 

0 

5 

5000000 

&  Combat  Package  System  2 

0 

0 

999 

200 

4 

0 

0 

4 

4000000 

Management  Package  System  3 

0 

0 

999 

300 

3 

0 

0 

3 

3000000 

Table  4  is  a  hypothetical  and  simplified  catalogue  of  individual  systems  available  to  the 


Navy  in  its  pursuit  of  achieving  desired  capabilities.  Although  the  numbers  are  hypothetical  and 


do  not  explicitly  illustrate  live  data,  the  salient  features  of  the  acquisition  problem  are  still 
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preserved.  Table  4  lists  systems  that  are  available  for  each  of  the  three  mission  packages — ASW, 
MCM,  SUW — along  with  an  individual  rating  of  system  capabilities  and  requirements  for  the 
systems  to  operate.  Additionally,  Table  4  provides  the  system  development  time  and  associated 
acquisition  costs.  Systems  that  are  unable  to  provide  a  particular  capability  (or  do  not  have  a 
particular  requirement)  have  a  zero  entry.  Although  the  sea  frame  is  typically  a  single  system, 
the  current  sample  problem  couples  the  sea  frame  with  battle  management  software  as  a  base 
system  that  provides  intra  system  capabilities.  The  development  of  these  systems  is  based  on  a 
projected  time  schedule  that  is  inherently  subject  to  overruns  and  risk.  This  element  is  captured 
in  the  covariance  matrix  shown  in  Table  5. 


Table  5.  System  Interdependency  and  Development  Risk  (Covariance) 


Variable  Depth 

Multi  Fen  Tow 

Lightweight  tow 

RAMCS  II 

ALMDS  (MH-60) 

N-LOS  Missiles 

Griffin  Missiles 

Package  System  1 

Package  System  2 

Package  System  3 

Variable  Depth 

0.1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Multi  Fen  Tow 

0 

0.6 

0 

0 

0 

0 

0 

0 

0.1 

0 

Lightweight  tow 

0 

0 

0.2 

0 

0 

0 

0 

0 

0 

0.2 

RAMCS  II 

0 

0 

0 

0.3 

0.1 

0 

0 

0 

0.2 

0 

ALMDS  (MH-60) 

0 

0 

0 

1 

0.1 

0 

0 

0 

0 

0.3 

N-LOS  Missiles 

0 

0 

0 

0 

0 

0.5 

0.2 

0 

0.1 

0 

Griffin  Missiles 

0 

0 

0 

0 

0 

0.2 

0.3 

0 

0 

0 

Package  System  1 

0 

0 

0 

0 

0 

0 

0 

0.5 

0 

0 

Package  System  2 

0 

0.1 

0 

0.2 

0 

0.1 

0 

0 

0.3 

0 

Package  System  3 

0 

0 

0.2 

0 

0.3 

0 

0 

0 

0 

0.2 

Table  5  shows  the  risk  and  interdependency  aspects  of  the  decision  process.  The  diagonal 
terms  represent  the  variance  (degree  of  deviation  from  expected  time)  in  development  time.  The 
off-diagonal  terms  are  the  variances  due  to  interdependencies  between  individual  systems  that 
have  commonly  developed  subsystems.  For  example,  since  the  N-LOS  and  Griffin  missile 
systems  are  both  developed  by  Northrop  Grumman,  it  is  conceivable  that  they  have  common 
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parts  or  undergo  similar  processes  in  development  and  manufacturing.  The  covariance  value  in 
the  cross  term  therefore  represents  joint  development  risk  for  the  two  systems. 

Estimation  of  these  quantities  can  come  directly  from  manufacturing  and  development  data. 
In  the  case  of  new  systems,  the  quantities  can  be  estimated  heuristically  using  basic  rules  similar 
to  those  used  in  project  management  techniques  such  as  PERT  and  other  CPM  methods.  The 
entries  of  the  matrix  in  Table  5  are  typically  inferred  from  data;  in  this  case,  the  values  are 
hypothetically  developed  for  the  concept  example  problem.  Most  of  the  individual  systems  do 
not  bear  many  interdependencies,  with  the  exception  of  the  sea  frame  and  combat  management 
support  systems  that  are  interlinked  more  explicitly  to  other  listed  systems  in  Table  5.. 
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Investment  Model  Formulation  and  Solution 

The  problem  statement  for  the  given  acquisition  problem  is  formulated  as  a  mathematical 
optimization  problem,  which  requires  the  definition  of  two  primary  segments.  The  objective 
function  is  the  equation  that  describes  the  primary  metric  to  be  optimized.  This  typically 
translates  to,  for  example,  the  maximization  of  profits  or  minimization  of  costs/risk  in  the 
commercial  sense.  The  second  important  aspect  deals  with  the  formulation  of  constraints,  which 
are  equations  that  typically  describe  resource  (e.g.,  time,  cost)  constraints  on  the  system  and  can 
be  manipulated  to  reflect  the  salient  conditions  of  the  problem  to  be  solved.  The  investment 
portfolio  problem  presented  in  the  formulation  shown  below  seeks  to  maximize  the  aggregate 
capabilities  of  an  SoS  architecture  while  minimizing  cost,  developmental  time,  and  integration 
risks.  The  mathematical  model  for  the  concept  problem  can  be  written  as  the  following: 

-w-x‘)  ~HX,F  f  hK  -  I(CX )]  (>9> 

V  9  V  ilc  7  9  y 

XB  c 

X't  =  — - — —  (Portfolio  Fractions)  (20) 

B 

+  e  =  B  (Budget  Constraint)  (21) 

^SqCX;  >  ^SqRX;  (Satisfy  All  System  Requirements)  (22) 

9  9 

X*  +  Xj  +  XBx  =  1  (ASW  System  Compatibility)  (23) 


Xj  +  XB  =  1  (MCM  System  Compatibility) 


(24) 
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X l  +  Xj  =  1  (SUW  System  Compatibility)  (25) 

Xj  +  Xj  +  Xf0  =  1  (Package  System  Compatibility)  (26) 

XBq  e  {0,1}  (binary)  (27) 

The  mathematical  model  shown  by  Equations  19-27  represents  the  formulation  of  a  traditional 
single-stage  optimization  problem  that  is  typical  of  operations  research  and  financial  engineering 
circles.  The  current  form  for  the  portfolio  model  at  hand  is  known  as  a  quadratic  integer  program 
(QIP)  and  is  based  on  the  Markowitz  formulation  that  seeks  to  generate  optimal  portfolios  that 
balance  potential  expected  rewards  against  risk.  Equation  19  is  the  objective  function — in  this 
equation,  the  objective  is  to  maximize  overall  capability  while  minimizing  cost  and  development 
risk.  Equation  20  is  the  fraction  of  the  budget  invested  in  individual  systems.  Equation  2 1  is  the 
budgeting  constraints,  where  the  sum  of  all  investments  in  individual  systems  (and  savings)  must 
be  equal  to  the  total  budget  allotted.  Equation  22  ensures  that  all  requirements  of  individual 
systems  must  be  met.  Equations  23-26  are  the  individual  system  compatibilities.  In  Equations 
23-26,  this  translates  to  the  selection  of  one  system  from  each  mission  package  (ASW,  MCM, 
SUW)  and  a  seaframe  and  combat  management  package  that  services  the  mission  modules. 

These  packages  are  mutually  exclusive  and  therefore  warrant  a  total  selection  of  summation 
equal  to  1,  which  ensures  that  no  two  packages  per  category  are  selected  to  satisfy  the  respective 
requirements.  The  covariance  matrix,  as  denoted  byZ;>. ,  represents  variations  in  development 

time  due  to  system  interdependencies.  The  27.  fonnulation  is  amenable  to  several  methods  of 

solution  using  both  freeware  and  commercially  available  solvers  that  are  written  with  system 
integration  and  IT  considerations  in  mind.  Models  using  these  solver  platforms  are  readily 
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integrated  into  IT  environments  and  enterprise  systems,  providing  a  model-centric  environment 
for  the  decision-making  process. 
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Preliminary  Results 


The  portfolio  optimization  problem  is  modeled  and  solved  using  a  commercial  solver  by 
varying  the  risk  aversion  parameter,  A  ,  to  generate  the  performance  efficiency  frontier.  By 
changing  this  parameter,  the  portfolio’s  aversion  to  risk  is  increased  as  the  penalty  effect  of  A  is 
more  pronounced  with  increasing  value.  The  increase  in  the  value  of  risk  aversion  forces  the 
portfolio  to  select  assets  (systems)  that  are  lower  risk,  which  consequently  results  in  a  lower  but 
less  risky  performance,  as  exhibited  in  Figure  5. 
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Figure  5.  Portfolio  Reward  to  Risk  Ratio 

Error!  Reference  source  not  found.Error!  Reference  source  not  found,  shows  what  is 
known  as  the  Markowitz  efficiency  frontier,  where  a  collection  of  optimal  portfolios  that 
maximizes  performance  are  shown  for  decreasing  values  of  risk  aversion.  Each  labeled  portfolio 
point  on  the  frontier  is  comprised  of  selected  systems,  as  shown  in  the  given  Figure  5.  In  a  more 
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realistic  setting,  Figure  5  can  be  used  to  explore  the  benefits  of  introducing  higher-performing, 
though  potentially  riskier,  systems  into  the  conglomeration.  For  instance,  the  highest-risk 
portfolio  includes  the  use  of  the  N-LOS  missile  system.  In  reality,  the  N-LOS  system  was 
extricated  from  the  LCS  program  due  to  poor  consistency  in  performance  and  replaced  by  the 
shorter-ranged  (and  cheaper)  but  more  reliable  Griffin  missile  system.  This  tradeoff  of 
performance  due  to  poor  consistency  is  exhibited  in  Figure  5,  where  reduced  performance  is 
complemented  by  reduced  risk.  The  results  show  the  potential  valuation  of  risk  taken  with  the 
introduction  of  each  system  for  increased  performance.  It  also  identifies  critical  systems  that  are 
common  across  the  performance  index,  such  as  the  Airborne  Laser  Mine  Detection  System 
(ALMDS)  that  is  a  fixed  asset  across  all  ranges  of  performance..  The  results  from  this  portfolio 
frontier  serve  as  a  complement  to  the  CEM  methodology;  the  CEM  can  be  used  to  explore 
inherent  network  dynamics  for  each  portfolio  that  exists  on  the  frontier. 
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Future  Work 


Robust  Portfolio  Method 

It  is  well  known  in  financial  engineering  circles  that  the  Markowitz  formulation,  as  used  in 
the  simplified  LCS  scenario,  is  sensitive  to  changes  in  estimated  quantities  of  the  covariance 
matrix  (system  interdependencies)  and  expected  return  (system  performance).  The  sensitivity 
due  to  poor  covariance  estimations  can  result  in  highly  inefficient  portfolios  due  to  errors  in 
estimation  or  market  shifts.  Such  sensitivity  issues  has  prompted  the  development  of  a  variety  of 
robust  methods  in  portfolio  analysis  to  ensure  that  the  chosen  portfolio  of  assets  are  stable 
against  potential  changes  in  market  conditions/expected  volatility. 

The  generalized  version  of  the  current  portfolio  formulation  as  dictated  in  Equations  19-27 
can  be  refonnulated  using  robust  optimization  techniques;  this  includes  Semi-Definite 
Programming  (SDP)  approaches  (Fabozzi  et  ah,  2007;  Tutuncu  &  Cornuejols,  2007)  that  are 
extensions  of  modern  portfolio  and  control  theory.  The  reformulation  allows  for  possible 
changes  in  estimated  quantities  (e.g.,  due  to  market  shifts  in  pricing,  volatility,  system 
interdependencies)  are  taken  into  consideration  more  explicitly  as  uncertainty  sets.  The  resulting 
portfolio  allocation  will  not  change  appreciably,  even  if  salient  estimated  quantities  or  benefits 
change  (within  prescribed  limits).  In  the  context  of  an  acquisition  problem,  the  use  of  a  robust 
formulation  translates  to  reduced  transaction  costs  associated  with  having  to  extricate  legacy 
systems  and  robustness  against  market  estimation  errors,  volatility,  and  changing  requirement 
conditions.  The  general  form  of  the  portfolio  problem  in  this  research  can  be  reposed  as  a 
robust,  SDP  problem,  as  given  by  Equation  28  (Fabozzi  et  al.,  2007): 
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max 
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,min{^.x}-A{(AE)-(AS>}j 
s.t. 


A- A 


x 

1 


±  0 


Ax  >  b 


Cx  =  d 
A  >  0,A  >  0 

us  (m)  =  {m  I  Mi  ~  Mi  1^  8»i  = 
Z<S<Z 


(28) 


Although  the  complexity  of  the  optimization  problem  increases,  it  is  nevertheless  very 
amenable  to  a  collection  of  numerical  methods  and  provides  good  performance  for  realistic 
portfolio  problems,  especially  under  highly  volatile  conditions  (Fabozzi  et  al.,  2007).  Equations 
29  and  30  in  the  following  section  represent  the  uncertainty  bounds  of  the  perfonnance 
(capability)  of  each  system  and  the  associated  risk  of  interdependencies. 


Multi-Period  Investment  Portfolio 


The  general  portfolio  formulation  in  this  section  considers  a  static  portfolio  approach 
without  consideration  for  sequential,  multi-period  investment  horizons.  An  addition  of  multi- 
step  considerations  into  the  decision  process  makes  the  problem  amenable  to  dynamic 
programming  and  control  theory  methods.  The  extension  of  the  investment  portfolio  approach 
generally  amounts  to  rewriting  the  objective  of  the  optimization  problem  as  the  following: 


max 


V  9  V 


R 


Vi  ~Z(C,X‘)  +£(4.,  I*w„ 


(29) 
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The  objective  function  now  reflects  consequential  effects  where  current  acquisition 
decisions  affect  latter  decisions,  as  denoted  by  the  expectation  term  of  Equation  29.  The 
stochastic  nature  of  the  mathematical  problem  in  Equation  29  is  a  stochastic  control  problem, 
which  has  a  variety  of  efficient  and  industry-tested  solutions,  such  as  approximate  dynamic 
programming  (ADP)  and  Model  Predictive  Control  (MPC).  Thus,  the  current  motivation  of 
extending  the  established  research  framework  is  to  bring  these  tools  (such  as  ADP  amd  MPC)  to 
bear  upon  the  immediate  acquisition  problem,  keeping  enterprise,  model-centric  architectures  of 
decision-making  processes  in  mind. 

Markov  Model  Extensions 

The  analytical  Markov  model  will  be  extended  to  include  dynamic  analysis  of  failure  modes 
that  are  associated  with  the  Markov  approach  presented  in  this  report.  The  extension  of  the 
Markov  approach  will  include  the  framework  of  identifying  delay  time  metrics  at  each  node  to 
the  analysis  of  eigenmodes  (connected  trees/networks)  of  disruption  sequences.  The 
identification  of  disruption  eigenmodes  is  particularly  useful  in  model-centric  enterprise  systems, 
where  clear  visualization  of  connectivity  between  individual  systems  is  crucial  to  the  design  and 
diagnosis  of  underlying  problems. 
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